[ N 2019-10-07 06:28:49.1135 125340/T5 age/Sha/ApiServerUtils.h:733 ]: All log file(s) reopened. [ N 2019-10-07 06:28:49.1171 125344/Tt7 age/Sha/ApiServerUtils.h:792 ]: All log file(s) reinherited. [ N 2019-10-07 06:29:02.5758 125340/T5 age/Sha/ApiServerUtils.h:733 ]: All log file(s) reopened. [ N 2019-10-07 06:29:02.5772 125344/Tt9 age/Sha/ApiServerUtils.h:792 ]: All log file(s) reinherited. App 125491 output: I, [2019-10-07T06:53:20.903787 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T07:34:50.927487 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125511 output: I, [2019-10-07T08:03:09.636324 #125511] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T08:21:58.625062 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125511 output: I, [2019-10-07T08:22:06.684657 #125511] INFO -- omniauth: (joshid) Callback phase initiated. App 125491 output: I, [2019-10-07T08:24:32.103000 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T08:24:42.979268 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T08:24:48.944859 #125491] INFO -- omniauth: (joshid) Callback phase initiated. App 125491 output: I, [2019-10-07T08:31:18.162668 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T08:31:23.510928 #125491] INFO -- omniauth: (joshid) Callback phase initiated. App 125491 output: I, [2019-10-07T08:32:54.711476 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125511 output: I, [2019-10-07T08:33:03.215940 #125511] INFO -- omniauth: (joshid) Callback phase initiated. App 125491 output: I, [2019-10-07T08:37:16.565461 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125511 output: I, [2019-10-07T08:49:47.719452 #125511] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: WARNING: terminating connection because of crash of another server process App 125491 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125491 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125491 output: [ 2019-10-07 09:13:09.4018 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::ConnectionBad: PQconsumeInput() SSL SYSCALL error: EOF detected App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:09.4531 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428041] Sending 502 response: application did not send a complete response App 125611 output: WARNING: terminating connection because of crash of another server process App 125611 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125611 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 4145 output: WARNING: terminating connection because of crash of another server process App 125591 output: WARNING: terminating connection because of crash of another server process App 4145 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125591 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 4145 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125591 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125531 output: WARNING: terminating connection because of crash of another server process App 125531 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125531 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125571 output: WARNING: terminating connection because of crash of another server process App 125571 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125571 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125551 output: WARNING: terminating connection because of crash of another server process App 125551 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125551 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 61001 output: WARNING: terminating connection because of crash of another server process App 61001 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 61001 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 114383 output: WARNING: terminating connection because of crash of another server process App 114383 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 114383 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125511 output: WARNING: terminating connection because of crash of another server process App 125511 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125511 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. [ W 2019-10-07 09:13:09.5670 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428042] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:09.5668 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:10.1806 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' [ W 2019-10-07 09:13:10.1808 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428042] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:10.3608 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1ql2sp46vj8gf19mn84lodqsok6kuqto9jmhvkyr1ydqy38pd4' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' [ W 2019-10-07 09:13:10.3609 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428042] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 4145 output: [ 2019-10-07 09:13:10.6254 4145/0x0000000002f765a0(Worker 1) utils.rb ]: *** Exception PG::Error in Rack application object (FATAL: the database system is in recovery mode App 4145 output: FATAL: the database system is in recovery mode App 4145 output: ) (process 4145, thread 0x0000000002f765a0(Worker 1)): App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `set_client_encoding' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `configure_connection' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:255:in `reconnect!' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:423:in `verify!' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:778:in `block in checkout_and_verify' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:126:in `call' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:455:in `call' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:101:in `__run_callbacks__' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:750:in `_run_checkout_callbacks' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:777:in `checkout_and_verify' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:128:in `retrieve_connection' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:91:in `connection' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:180:in `replace_bind_variables' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:169:in `sanitize_sql_array' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:33:in `sanitize_sql_for_conditions' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/where_clause_factory.rb:14:in `build' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:632:in `where!' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:625:in `where' App 4145 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:224:in `validate' App 4145 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 4145 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 4145 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 4145 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 4145 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 4145 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 4145 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 4145 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 4145 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125591 output: [ 2019-10-07 09:13:10.6272 125591/0x000000000311c3f0(Worker 1) utils.rb ]: *** Exception PG::Error in Rack application object (FATAL: the database system is in recovery mode App 125591 output: FATAL: the database system is in recovery mode App 125591 output: ) (process 125591, thread 0x000000000311c3f0(Worker 1)): App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `set_client_encoding' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `configure_connection' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:255:in `reconnect!' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:423:in `verify!' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:778:in `block in checkout_and_verify' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:126:in `call' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:455:in `call' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:101:in `__run_callbacks__' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:750:in `_run_checkout_callbacks' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:777:in `checkout_and_verify' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:128:in `retrieve_connection' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:91:in `connection' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:180:in `replace_bind_variables' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:169:in `sanitize_sql_array' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:33:in `sanitize_sql_for_conditions' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/where_clause_factory.rb:14:in `build' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:632:in `where!' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:625:in `where' App 125591 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:224:in `validate' App 125591 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125591 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125591 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125591 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125591 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125591 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' [ W 2019-10-07 09:13:10.6273 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428041] Sending 502 response: application did not send a complete response App 125591 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125591 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125591 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125611 output: [ 2019-10-07 09:13:10.6289 125611/0x000000000311c3f0(Worker 1) utils.rb ]: *** Exception PG::Error in Rack application object (FATAL: the database system is in recovery mode App 125611 output: FATAL: the database system is in recovery mode App 125611 output: ) (process 125611, thread 0x000000000311c3f0(Worker 1)): App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `set_client_encoding' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `configure_connection' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:255:in `reconnect!' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:423:in `verify!' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:778:in `block in checkout_and_verify' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:126:in `call' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:455:in `call' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:101:in `__run_callbacks__' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:750:in `_run_checkout_callbacks' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:777:in `checkout_and_verify' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:128:in `retrieve_connection' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:91:in `connection' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:180:in `replace_bind_variables' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:169:in `sanitize_sql_array' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:33:in `sanitize_sql_for_conditions' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/where_clause_factory.rb:14:in `build' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:632:in `where!' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:625:in `where' App 125611 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:224:in `validate' App 125611 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125611 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125611 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125611 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125611 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125611 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125611 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125611 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125611 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 61001 output: [ 2019-10-07 09:13:10.6300 61001/0x0000000002892888(Worker 1) utils.rb ]: *** Exception PG::Error in Rack application object (FATAL: the database system is in recovery mode App 61001 output: FATAL: the database system is in recovery mode App 61001 output: ) (process 61001, thread 0x0000000002892888(Worker 1)): App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `set_client_encoding' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `configure_connection' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:255:in `reconnect!' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:423:in `verify!' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:778:in `block in checkout_and_verify' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:126:in `call' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:455:in `call' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:101:in `__run_callbacks__' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:750:in `_run_checkout_callbacks' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:777:in `checkout_and_verify' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:128:in `retrieve_connection' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:91:in `connection' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:180:in `replace_bind_variables' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:169:in `sanitize_sql_array' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:33:in `sanitize_sql_for_conditions' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/where_clause_factory.rb:14:in `build' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:632:in `where!' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:625:in `where' App 61001 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:224:in `validate' App 61001 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 61001 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 61001 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 61001 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 61001 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 61001 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 61001 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 61001 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 61001 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 114383 output: [ 2019-10-07 09:13:10.6321 114383/0x0000000002ad2418(Worker 1) utils.rb ]: *** Exception PG::Error in Rack application object (FATAL: the database system is in recovery mode App 114383 output: FATAL: the database system is in recovery mode App 114383 output: ) (process 114383, thread 0x0000000002ad2418(Worker 1)): App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `set_client_encoding' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `configure_connection' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:255:in `reconnect!' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:423:in `verify!' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:778:in `block in checkout_and_verify' [ W 2019-10-07 09:13:10.6323 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428042] Sending 502 response: application did not send a complete response App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:126:in `call' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:455:in `call' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:101:in `__run_callbacks__' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:750:in `_run_checkout_callbacks' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:777:in `checkout_and_verify' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:128:in `retrieve_connection' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:91:in `connection' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:180:in `replace_bind_variables' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:169:in `sanitize_sql_array' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:33:in `sanitize_sql_for_conditions' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/where_clause_factory.rb:14:in `build' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:632:in `where!' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:625:in `where' App 114383 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:224:in `validate' App 114383 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 114383 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 114383 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 114383 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 114383 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 114383 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 114383 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 114383 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 114383 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125571 output: [ 2019-10-07 09:13:10.6387 125571/0x000000000311c418(Worker 1) utils.rb ]: *** Exception PG::Error in Rack application object (FATAL: the database system is in recovery mode App 125571 output: FATAL: the database system is in recovery mode App 125571 output: ) (process 125571, thread 0x000000000311c418(Worker 1)): App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `set_client_encoding' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `configure_connection' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:255:in `reconnect!' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:423:in `verify!' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:778:in `block in checkout_and_verify' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:126:in `call' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:455:in `call' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:101:in `__run_callbacks__' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:750:in `_run_checkout_callbacks' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:777:in `checkout_and_verify' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:128:in `retrieve_connection' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:91:in `connection' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:180:in `replace_bind_variables' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:169:in `sanitize_sql_array' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:33:in `sanitize_sql_for_conditions' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/where_clause_factory.rb:14:in `build' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:632:in `where!' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:625:in `where' App 125571 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:224:in `validate' App 125571 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125571 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125571 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125571 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125571 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125571 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125571 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125571 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125571 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125511 output: [ 2019-10-07 09:13:10.6468 125511/0x0000000003133f78(Worker 1) utils.rb ]: *** Exception PG::Error in Rack application object (FATAL: the database system is in recovery mode App 125511 output: FATAL: the database system is in recovery mode App 125511 output: ) (process 125511, thread 0x0000000003133f78(Worker 1)): App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `set_client_encoding' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `configure_connection' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:255:in `reconnect!' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:423:in `verify!' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:778:in `block in checkout_and_verify' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:126:in `call' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:455:in `call' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:101:in `__run_callbacks__' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:750:in `_run_checkout_callbacks' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:777:in `checkout_and_verify' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:128:in `retrieve_connection' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:91:in `connection' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:180:in `replace_bind_variables' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:169:in `sanitize_sql_array' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:33:in `sanitize_sql_for_conditions' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/where_clause_factory.rb:14:in `build' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:632:in `where!' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:625:in `where' App 125511 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:224:in `validate' App 125511 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125511 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125511 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125511 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125511 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125511 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125511 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125511 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125511 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125531 output: [ 2019-10-07 09:13:10.6519 125531/0x000000000311c3f0(Worker 1) utils.rb ]: *** Exception PG::Error in Rack application object (FATAL: the database system is in recovery mode App 125531 output: FATAL: the database system is in recovery mode App 125531 output: ) (process 125531, thread 0x000000000311c3f0(Worker 1)): App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `set_client_encoding' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `configure_connection' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:255:in `reconnect!' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:423:in `verify!' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:778:in `block in checkout_and_verify' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:126:in `call' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:455:in `call' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:101:in `__run_callbacks__' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:750:in `_run_checkout_callbacks' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:777:in `checkout_and_verify' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:128:in `retrieve_connection' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:91:in `connection' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:180:in `replace_bind_variables' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:169:in `sanitize_sql_array' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/sanitization.rb:33:in `sanitize_sql_for_conditions' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/where_clause_factory.rb:14:in `build' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:632:in `where!' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:625:in `where' App 125531 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:224:in `validate' App 125531 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125531 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125531 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125531 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125531 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125531 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125531 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125531 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125531 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125551 output: [ 2019-10-07 09:13:10.6540 125551/0x000000000311c5d0(Worker 1) utils.rb ]: *** Exception PG::Error in Rack application object (FATAL: the database system is in recovery mode App 125551 output: FATAL: the database system is in recovery mode App 125551 output: ) (process 125551, thread 0x000000000311c5d0(Worker 1)): App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `set_client_encoding' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `configure_connection' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:255:in `reconnect!' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:423:in `verify!' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:778:in `block in checkout_and_verify' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:126:in `call' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:455:in `call' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:101:in `__run_callbacks__' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:750:in `_run_checkout_callbacks' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:777:in `checkout_and_verify' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:128:in `retrieve_connection' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_handling.rb:91:in `connection' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/delegation.rb:43:in `connection' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/query_methods.rb:105:in `bound_attributes' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:13:10.6542 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428041] Sending 502 response: application did not send a complete response App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125551 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125551 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125551 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125551 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125551 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125551 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125551 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125551 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125551 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125551 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:10.6787 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' [ W 2019-10-07 09:13:10.6788 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428042] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:11.1772 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:11.1775 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428042] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:11.3771 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' [ W 2019-10-07 09:13:11.3773 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428043] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:11.4583 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428043] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:11.4579 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:11.7269 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:12.1762 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' [ W 2019-10-07 09:13:12.1764 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428043] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:12.6407 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:12.6608 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:12.6644 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125511 output: WARNING: terminating connection because of crash of another server process App 125511 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125511 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125491 output: [ 2019-10-07 09:13:12.6674 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125531 output: WARNING: terminating connection because of crash of another server process App 125531 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125531 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125491 output: [ 2019-10-07 09:13:13.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' [ W 2019-10-07 09:13:13.1759 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428044] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:13.5870 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' [ W 2019-10-07 09:13:13.5872 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428044] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:13.7391 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:14.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' [ W 2019-10-07 09:13:14.1764 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428044] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:14.6659 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:14.6736 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:14.6760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:14.6780 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:15.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' [ W 2019-10-07 09:13:15.1757 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428045] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:15.7509 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:15.7972 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:16.1401 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:16.1752 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428045] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:16.1751 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:16.6826 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:16.6878 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:16.6895 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:17.1761 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' [ W 2019-10-07 09:13:17.1763 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428046] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:17.2689 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:17.7635 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:17.8118 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:18.1522 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:18.1751 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' [ W 2019-10-07 09:13:18.1753 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428047] Sending 502 response: application did not send a complete response App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:18.6952 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:18.6997 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:18.7695 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:18.8110 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' [ W 2019-10-07 09:13:18.8111 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428047] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:19.1780 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' [ W 2019-10-07 09:13:19.1781 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428047] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:19.2804 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:19.7751 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:19.8231 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:20.1684 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:20.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' [ W 2019-10-07 09:13:20.1764 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428048] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:20.5549 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:13:20.5550 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428048] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:20.7107 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:20.7134 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:20.7810 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:21.1761 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' [ W 2019-10-07 09:13:21.1763 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428049] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:21.2918 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:21.3315 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:21.3317 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428049] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:21.3409 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' [ W 2019-10-07 09:13:21.3410 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428049] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:21.7861 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:21.7982 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:21.8357 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:22.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' [ W 2019-10-07 09:13:22.1758 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428049] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:22.1784 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:22.7246 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:22.7263 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:22.7920 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:23.1002 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' [ W 2019-10-07 09:13:23.1004 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428050] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:23.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' [ W 2019-10-07 09:13:23.1756 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428050] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:23.3025 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:23.7976 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:23.8080 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:23.8476 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:24.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:24.1765 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428051] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:24.1889 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:24.2683 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:24.2685 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428051] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:24.7369 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:24.8063 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:24.8086 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:25.0755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' [ W 2019-10-07 09:13:25.0757 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428052] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:25.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:13:25.1759 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428052] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:25.3155 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:25.7481 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:13:25.7482 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428052] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:25.8096 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:25.8199 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:25.8394 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:25.8582 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:26.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' [ W 2019-10-07 09:13:26.1759 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428053] Sending 502 response: application did not send a complete response App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:26.1990 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:26.7468 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='340qsyeqr8w8qpvglf38vkyh25fouicx7k9avp5x6voxszp2d1' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' [ W 2019-10-07 09:13:26.7470 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428053] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:26.7485 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:26.7925 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428053] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:26.7921 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='340qsyeqr8w8qpvglf38vkyh25fouicx7k9avp5x6voxszp2d1' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:26.8168 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:26.8187 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:27.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' [ W 2019-10-07 09:13:27.1757 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428054] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:27.3272 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:27.8201 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:27.8325 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:27.8496 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:27.8674 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:28.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' [ W 2019-10-07 09:13:28.1756 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428054] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:28.2095 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:28.7556 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:28.8291 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:29.0525 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' [ W 2019-10-07 09:13:29.0527 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428055] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:29.0640 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' [ W 2019-10-07 09:13:29.0642 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428055] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:29.1756 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428055] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:29.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:29.3380 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:29.8270 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' [ W 2019-10-07 09:13:29.8272 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428055] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:29.8316 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:29.8434 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:29.8607 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:29.8810 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:30.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:13:30.1765 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428056] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:30.2203 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:30.7678 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:30.8279 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:13:30.8280 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428056] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:30.8390 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:31.1757 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428057] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:31.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:31.3477 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:31.8440 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:31.8553 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:31.8710 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:31.8916 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:32.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' [ W 2019-10-07 09:13:32.1764 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428057] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:32.2328 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:32.6391 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' [ W 2019-10-07 09:13:32.6392 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428058] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:32.7803 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:32.8513 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:33.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:33.1755 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428058] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:33.3583 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:33.8553 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:33.8677 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:33.8833 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:33.8969 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:34.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' [ W 2019-10-07 09:13:34.1755 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428059] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:34.2373 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:34.7928 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:34.7950 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:34.8629 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:35.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' [ W 2019-10-07 09:13:35.1758 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428059] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:35.5692 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:35.8660 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:35.8666 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428060] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:35.8821 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:35.8935 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:35.9051 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): [ W 2019-10-07 09:13:35.9053 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428060] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:35.9073 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:35.9334 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:36.1762 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:13:36.1763 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428061] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:36.2486 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:36.8057 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:36.8683 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:37.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:13:37.1758 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428061] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:37.5822 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:37.8982 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:37.9071 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:37.9223 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:37.9454 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:38.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:13:38.1756 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428062] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:38.2598 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:38.8180 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:38.8829 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:39.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:39.1762 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428062] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:39.5933 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:39.9040 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:39.9179 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:39.9329 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:39.9625 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:40.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' [ W 2019-10-07 09:13:40.1761 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428063] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:40.2721 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:40.8293 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:40.8949 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:41.0709 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' [ W 2019-10-07 09:13:41.0711 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428064] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:41.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:13:41.1764 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428064] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:41.6036 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:41.9178 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:41.9282 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:41.9437 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:41.9734 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:42.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:42.1762 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428065] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:42.2845 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:42.8414 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:42.9081 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:43.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:13:43.1756 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428065] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:43.3611 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "containers".* FROM "containers" WHERE "containers"."uuid" = $1 ORDER BY "containers"."id" ASC LIMIT $2) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `async_exec_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:608:in `block in exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:607:in `exec_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:591:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in `select_prepared' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:39:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' [ W 2019-10-07 09:13:43.3613 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428065] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:109:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:43.6139 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:43.9301 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:43.9329 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:43.9539 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:43.9846 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:44.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' [ W 2019-10-07 09:13:44.1757 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428066] Sending 502 response: application did not send a complete response App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:44.2949 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:44.7927 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:44.8566 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:44.9210 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:45.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' [ W 2019-10-07 09:13:45.1762 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428067] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:45.6248 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:45.9429 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:45.9443 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:45.9634 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:45.9961 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:46.1673 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' [ W 2019-10-07 09:13:46.1675 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428067] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:46.1802 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:46.1804 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428067] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:46.3057 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:46.8045 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:46.8742 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:46.9331 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:47.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:13:47.1756 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428068] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:47.6358 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:47.9544 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:47.9560 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:47.9726 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:48.0055 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:48.1753 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' [ W 2019-10-07 09:13:48.1755 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428069] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:48.3168 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:48.8165 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:48.8863 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:48.9459 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:49.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:13:49.1757 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428069] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:49.6465 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:49.9663 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:49.9694 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:49.9819 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:50.0181 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:50.1756 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428070] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:50.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:50.3277 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:50.8281 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:50.9003 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:50.9573 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:51.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:51.1759 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428071] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:51.2392 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' [ W 2019-10-07 09:13:51.2394 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428071] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:51.6582 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:51.9794 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:51.9842 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:51.9869 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:52.0294 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:52.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' [ W 2019-10-07 09:13:52.1760 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428072] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:52.3330 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:52.8409 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:52.9118 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:52.9672 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:53.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' [ W 2019-10-07 09:13:53.1756 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428072] Sending 502 response: application did not send a complete response App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:53.9842 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:53.9972 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:53.9986 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:54.0346 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:54.1752 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:13:54.1754 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428073] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:54.3553 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:54.8521 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:54.9232 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:54.9722 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:55.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:13:55.1756 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428074] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:55.9949 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:56.0073 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:56.0092 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:56.0447 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:56.1753 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:56.1755 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428074] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:56.2847 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:56.2850 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428074] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:56.3659 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:56.8657 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:56.9400 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:56.9861 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:57.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:57.1762 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428075] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:58.0046 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:58.0128 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:58.0183 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:58.0550 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:58.1751 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428076] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:58.1750 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:58.3820 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:58.8771 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:58.9619 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:58.9989 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:13:59.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:13:59.1757 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428076] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:00.0155 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:00.0252 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:00.0283 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:00.0670 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:00.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' [ W 2019-10-07 09:14:00.1759 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428077] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:00.3923 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:00.8885 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:00.9732 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:01.0092 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:01.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:14:01.1758 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428077] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:01.3966 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' [ W 2019-10-07 09:14:01.3967 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428078] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:02.0232 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:02.0373 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:02.0392 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:02.0718 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1ql2sp46vj8gf19mn84lodqsok6kuqto9jmhvkyr1ydqy38pd4' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' [ W 2019-10-07 09:14:02.0720 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428078] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:02.0754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1ql2sp46vj8gf19mn84lodqsok6kuqto9jmhvkyr1ydqy38pd4' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:02.0758 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428078] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:02.1506 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:02.1749 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428079] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:02.1748 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:02.4049 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:02.4519 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1blpt9ue933xp8kydkcplvy01p2meorof6l1atqhn9udsnaohe' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): [ W 2019-10-07 09:14:02.4521 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428079] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:02.4660 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1blpt9ue933xp8kydkcplvy01p2meorof6l1atqhn9udsnaohe' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:14:02.4662 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428079] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:02.8945 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:02.9903 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:03.0313 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:03.1739 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1twd101o0d4dzk9en4fsv5hi8q3g9zsb9lm4zy7e87a6kc4akf' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' [ W 2019-10-07 09:14:03.1741 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428079] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:03.2022 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428080] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:03.2021 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1twd101o0d4dzk9en4fsv5hi8q3g9zsb9lm4zy7e87a6kc4akf' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:03.2621 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:14:03.2623 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428080] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:04.0344 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:04.0489 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:04.1203 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:04.1605 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:04.1747 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' [ W 2019-10-07 09:14:04.1748 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428081] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:04.4210 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:04.9066 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:04.9974 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:05.0432 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:05.1756 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428081] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:05.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:06.0516 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:06.0615 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:06.1314 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:06.1711 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:06.1745 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' [ W 2019-10-07 09:14:06.1746 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428082] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:06.4328 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:06.5028 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' [ W 2019-10-07 09:14:06.5030 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428082] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:06.7478 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='340qsyeqr8w8qpvglf38vkyh25fouicx7k9avp5x6voxszp2d1' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' [ W 2019-10-07 09:14:06.7480 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428082] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:06.7869 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428082] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:06.7867 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='340qsyeqr8w8qpvglf38vkyh25fouicx7k9avp5x6voxszp2d1' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:06.9171 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:07.0117 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:07.0538 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:07.1752 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:07.1755 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428083] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:08.0635 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:08.0719 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:08.1411 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:08.1748 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:08.1750 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428083] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:08.1821 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:08.4493 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:08.9285 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:09.0234 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:09.0644 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:09.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:09.1762 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428084] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:10.0746 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:10.0836 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:10.1558 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:10.1751 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:10.1753 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428085] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:10.1932 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:10.4548 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:10.9414 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:11.0357 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:11.0765 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:11.1779 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:11.1783 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428085] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:11.5677 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:14:11.5679 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428085] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:12.0855 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:12.0964 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:12.1674 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:12.1749 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:12.1751 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428086] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:12.2029 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:12.4658 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:12.9541 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:13.0492 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:13.0819 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:13.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:13.1756 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428087] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:14.0966 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:14.1090 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:14.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:14.1762 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428087] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:14.1791 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:14.2140 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:14.4769 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:14.9656 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:15.0618 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:15.0936 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:15.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:15.1760 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428088] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:16.1092 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:16.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:16.1758 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428088] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:16.1965 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:16.2180 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:16.4877 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:16.6289 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:16.6291 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428089] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:16.9770 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:17.0730 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:17.1046 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:17.1752 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:17.1754 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428089] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:18.1203 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:18.1748 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' [ W 2019-10-07 09:14:18.1750 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428089] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:18.2060 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:18.2298 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:18.4991 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:18.9884 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:19.0866 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:19.1167 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:19.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' [ W 2019-10-07 09:14:19.1761 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428090] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:20.1400 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:20.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:14:20.1759 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428091] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:20.2279 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:20.2362 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:20.5136 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:20.9953 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:21.1003 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:21.1279 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:21.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:14:21.1757 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428091] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:21.7277 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428092] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:21.7275 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:22.1520 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:22.1750 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:14:22.1751 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428092] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:22.2326 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:22.2470 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:22.5248 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:23.0081 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:23.1062 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:23.1395 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:23.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' [ W 2019-10-07 09:14:23.1756 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428093] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:24.1641 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:24.1748 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' [ W 2019-10-07 09:14:24.1749 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428093] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:24.2429 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:24.2586 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:24.5371 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:25.0219 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:25.1239 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:25.1524 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:25.1753 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' [ W 2019-10-07 09:14:25.1755 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428094] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:26.1758 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428094] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:26.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:26.2557 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:26.2835 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:26.5483 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:26.8328 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:14:26.8330 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428095] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:27.0350 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:27.1357 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:27.1639 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:27.1748 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' [ W 2019-10-07 09:14:27.1750 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428095] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:28.1762 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428095] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:28.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:28.2677 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:28.2949 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:28.5536 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:29.0491 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:29.1524 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:29.1752 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:29.1972 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:29.1973 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428096] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:30.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' [ W 2019-10-07 09:14:30.1760 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428096] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:30.2786 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:30.3049 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:30.5653 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:31.0639 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:31.1654 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:31.1746 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:31.1748 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428097] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:31.1798 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:31.8813 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:31.8814 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428097] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:32.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' [ W 2019-10-07 09:14:32.1755 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428097] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:32.2888 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:32.3158 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:32.5772 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:33.0815 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:33.1764 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' [ W 2019-10-07 09:14:33.1765 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428098] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:33.1907 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:33.2368 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:34.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' [ W 2019-10-07 09:14:34.1759 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428099] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:34.2991 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:34.3280 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:34.5884 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:35.0943 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:35.1764 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' [ W 2019-10-07 09:14:35.1765 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428099] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:35.2040 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:35.2485 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:35.8412 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:36.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' [ W 2019-10-07 09:14:36.1759 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428100] Sending 502 response: application did not send a complete response App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:36.3188 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:36.3380 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:36.6074 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:36.9590 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' [ W 2019-10-07 09:14:36.9592 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428100] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:37.1081 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:37.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' [ W 2019-10-07 09:14:37.1758 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428100] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:37.2152 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:37.2629 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:37.8524 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:38.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' [ W 2019-10-07 09:14:38.1759 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428101] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:38.3304 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:38.3491 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:38.6193 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:39.1150 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:39.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' [ W 2019-10-07 09:14:39.1756 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428102] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:39.2261 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:39.2763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:39.8636 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:40.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' [ W 2019-10-07 09:14:40.1761 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428102] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:40.3364 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:40.3598 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:40.6320 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:41.1267 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:41.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' [ W 2019-10-07 09:14:41.1758 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428103] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:41.2374 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:41.2873 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:41.8745 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:42.1249 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' [ W 2019-10-07 09:14:42.1251 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428103] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:42.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:14:42.1759 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428103] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:42.3481 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:42.3718 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:42.6436 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:43.1437 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:43.1750 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:14:43.1752 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428104] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:43.2514 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:43.2997 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:43.8858 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:44.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' [ W 2019-10-07 09:14:44.1760 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428104] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:44.3594 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:44.3875 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:44.6545 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:45.1560 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:45.1750 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:14:45.1751 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428105] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:45.2644 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:45.3110 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:45.8973 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:46.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' [ W 2019-10-07 09:14:46.1757 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428106] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:46.3708 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:46.3981 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:46.6600 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:47.1695 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:47.1748 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' [ W 2019-10-07 09:14:47.1750 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428106] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:47.1887 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' [ W 2019-10-07 09:14:47.1888 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428106] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:47.2789 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:47.3249 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:47.9099 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:48.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:14:48.1758 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428107] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:48.3818 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:48.4092 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:48.6720 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:49.1776 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' [ W 2019-10-07 09:14:49.1778 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428107] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:49.1814 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:49.2854 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:49.3371 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:49.9217 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:50.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' [ W 2019-10-07 09:14:50.1758 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428108] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:50.3931 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:50.4208 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:50.6837 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:51.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' [ W 2019-10-07 09:14:51.1757 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428108] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:51.1934 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:51.3007 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:51.3502 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:51.9322 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:52.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' [ W 2019-10-07 09:14:52.1758 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428109] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:52.2565 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' [ W 2019-10-07 09:14:52.2567 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428109] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:52.4048 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:52.4337 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:52.6953 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:53.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' [ W 2019-10-07 09:14:53.1758 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428110] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:53.2041 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:53.3109 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:53.3635 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:53.9378 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:54.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:14:54.1764 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428110] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:14:54.4158 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:54.4454 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:54.7090 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:55.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' [ W 2019-10-07 09:14:55.1760 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428111] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:55.2156 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:55.3254 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:55.3780 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:55.9492 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:56.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' [ W 2019-10-07 09:14:56.1756 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428111] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:56.4278 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:56.4512 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:56.7213 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:57.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' [ W 2019-10-07 09:14:57.1757 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428112] Sending 502 response: application did not send a complete response App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:57.2420 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:57.3357 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:57.3521 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' [ W 2019-10-07 09:14:57.3522 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428112] Sending 502 response: application did not send a complete response App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:57.3920 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:57.9602 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:58.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' [ W 2019-10-07 09:14:58.1758 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428113] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:58.4338 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:58.4638 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:58.7332 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:59.1761 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:14:59.1763 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428113] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:59.2573 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:59.3469 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:59.4047 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:14:59.9721 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:00.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' [ W 2019-10-07 09:15:00.1757 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428114] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:00.4465 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:00.4751 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:00.7449 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:01.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' [ W 2019-10-07 09:15:01.1765 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428114] Sending 502 response: application did not send a complete response App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:01.2707 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:01.3624 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:01.4159 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:01.9835 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:02.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:15:02.1758 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428115] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:02.4425 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' [ W 2019-10-07 09:15:02.4427 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428115] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:02.4602 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:02.4858 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:02.7216 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1ql2sp46vj8gf19mn84lodqsok6kuqto9jmhvkyr1ydqy38pd4' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' [ W 2019-10-07 09:15:02.7218 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428115] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:02.7566 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:03.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' [ W 2019-10-07 09:15:03.1761 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428116] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:03.2855 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:03.3866 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:03.4314 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:04.0001 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:04.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:15:04.1759 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428116] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:04.2250 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1ql2sp46vj8gf19mn84lodqsok6kuqto9jmhvkyr1ydqy38pd4' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' [ W 2019-10-07 09:15:04.2251 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428116] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:04.4714 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:04.4989 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:04.7619 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:05.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:15:05.1757 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428117] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:05.2982 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:05.3993 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:05.4455 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:06.0109 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:06.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' [ W 2019-10-07 09:15:06.1761 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428118] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:06.4827 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:06.5118 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:06.7725 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:07.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:15:07.1762 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428118] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:07.3108 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:07.4066 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:07.4581 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:07.5198 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' [ W 2019-10-07 09:15:07.5200 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428119] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:08.0233 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:08.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:08.1759 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428119] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:08.4954 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:08.5231 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:08.7840 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:09.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:09.1761 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428119] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:09.3234 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:09.4179 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:09.4647 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:10.0357 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:10.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' [ W 2019-10-07 09:15:10.1757 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428120] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:10.5069 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:10.5341 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:10.7949 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:11.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:15:11.1761 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428120] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:11.3366 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:11.4311 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:11.4770 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:12.0412 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:12.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' [ W 2019-10-07 09:15:12.1765 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428121] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:12.5203 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:12.5447 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:12.6250 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:15:12.6252 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428121] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:12.8063 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:13.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): [ W 2019-10-07 09:15:13.1758 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428122] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:13.3494 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:13.4425 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:13.4881 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:14.0563 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:14.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:15:14.1760 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428122] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:14.5318 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:14.5572 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:14.8175 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:15.1762 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:15:15.1763 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428123] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:15.3560 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:15.4569 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:15.5011 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:16.0681 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:16.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' [ W 2019-10-07 09:15:16.1761 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428123] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:16.5375 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:16.5697 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:16.8293 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:17.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' [ W 2019-10-07 09:15:17.1765 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428124] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:17.3680 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:17.4694 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:17.5163 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:17.6990 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' [ W 2019-10-07 09:15:17.6992 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428124] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:18.0802 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:18.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:15:18.1758 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428125] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:18.5499 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:18.5795 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:18.8405 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:19.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' [ W 2019-10-07 09:15:19.1762 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428125] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:19.3818 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:19.4818 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:19.5285 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:20.0936 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:20.1768 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428126] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:20.1767 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:20.5627 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:20.5902 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:20.8519 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:21.1762 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428126] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:21.1761 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:21.3933 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:21.4929 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:21.5414 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:22.1071 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:22.1762 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' [ W 2019-10-07 09:15:22.1764 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428127] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:22.5740 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:22.6017 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:22.8160 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428127] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:22.8159 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:22.8629 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:23.1761 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:15:23.1763 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428128] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:23.4053 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:23.5036 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:23.5554 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:24.1177 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:24.1800 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' [ W 2019-10-07 09:15:24.1802 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428128] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:24.5868 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:24.6116 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:24.8776 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:25.1763 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428129] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:25.1761 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:25.4187 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:25.5090 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:25.5679 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:26.1319 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:26.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' [ W 2019-10-07 09:15:26.1761 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428129] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:26.5975 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:26.6242 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:26.7661 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:15:26.7663 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428130] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='340qsyeqr8w8qpvglf38vkyh25fouicx7k9avp5x6voxszp2d1' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:26.8010 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='340qsyeqr8w8qpvglf38vkyh25fouicx7k9avp5x6voxszp2d1' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): [ W 2019-10-07 09:15:26.8011 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428130] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:26.8888 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:27.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' [ W 2019-10-07 09:15:27.1760 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428130] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:27.4327 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:27.5222 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:27.5748 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:27.9221 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' [ W 2019-10-07 09:15:27.9223 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428131] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:28.1438 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:28.1758 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428131] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:28.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:28.6086 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:28.6373 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:28.9015 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:29.1765 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' [ W 2019-10-07 09:15:29.1766 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428131] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:29.4470 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:29.5321 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:29.5857 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:30.1493 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:30.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' [ W 2019-10-07 09:15:30.1761 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428132] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:30.6210 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:30.6476 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:30.9131 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:31.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' [ W 2019-10-07 09:15:31.1762 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428132] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:31.4616 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:31.5455 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:31.6003 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:32.1605 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:32.1752 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' [ W 2019-10-07 09:15:32.1754 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428133] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:32.6328 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:32.6537 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:32.9252 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:33.0402 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' [ W 2019-10-07 09:15:33.0404 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428134] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:33.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' [ W 2019-10-07 09:15:33.1762 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428134] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:33.4673 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:33.5562 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:33.6118 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:34.1720 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:34.1751 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): [ W 2019-10-07 09:15:34.1752 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428134] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:34.6391 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:34.6647 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:34.9369 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:35.1764 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:15:35.1765 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428135] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:35.4792 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:35.5688 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:35.6256 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:36.1764 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428135] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:36.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:36.1826 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:36.6505 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:36.6749 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:36.9482 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:37.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' [ W 2019-10-07 09:15:37.1758 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428136] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:37.4913 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:37.5791 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:37.6460 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:38.1300 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' [ W 2019-10-07 09:15:38.1302 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428136] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:38.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' [ W 2019-10-07 09:15:38.1758 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428137] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:38.1935 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:38.6625 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:38.6873 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:38.9598 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:39.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' [ W 2019-10-07 09:15:39.1761 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428137] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:39.5021 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:39.5904 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:39.6579 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:40.1764 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428138] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:40.1762 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:40.2185 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:40.6739 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:40.7027 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:40.9660 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:41.1771 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' [ W 2019-10-07 09:15:41.1773 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428138] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:41.5144 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:41.6008 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:41.6702 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:42.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' [ W 2019-10-07 09:15:42.1762 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428139] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:42.2295 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:42.6846 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:42.7133 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:42.9767 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:43.1761 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:15:43.1763 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428139] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:43.2920 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:43.2922 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428140] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:43.5263 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:43.6054 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:43.6820 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:44.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' [ W 2019-10-07 09:15:44.1764 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428140] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:44.2407 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:44.6976 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:44.7244 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:45.0040 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:45.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' [ W 2019-10-07 09:15:45.1763 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428141] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:45.5386 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:45.6171 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:45.6901 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:46.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' [ W 2019-10-07 09:15:46.1761 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428141] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:46.2507 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:46.7091 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:46.7355 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:47.0153 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:47.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' [ W 2019-10-07 09:15:47.1766 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428142] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:47.5511 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:47.6280 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:47.7013 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:48.1764 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:15:48.1766 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428142] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:48.2555 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:48.3303 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:15:48.3304 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428143] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:48.7204 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:48.7457 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:49.0267 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:49.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' [ W 2019-10-07 09:15:49.1761 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428143] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:49.5643 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:49.6407 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:49.7128 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:50.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' [ W 2019-10-07 09:15:50.1761 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428144] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:50.2668 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:50.7319 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:50.7574 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:51.0380 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:51.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' [ W 2019-10-07 09:15:51.1762 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428144] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:51.5713 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:51.6768 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:51.7658 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:52.1766 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' [ W 2019-10-07 09:15:52.1767 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428145] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:52.2808 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:52.7379 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:52.7693 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:53.0508 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:53.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:53.1761 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428145] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:53.4976 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:53.4979 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428145] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:53.5853 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:53.6885 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:53.7816 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:54.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:54.1762 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428146] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:54.2935 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:54.7499 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:54.7801 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:55.0652 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:55.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' [ W 2019-10-07 09:15:55.1761 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428147] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:55.5980 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:55.7003 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:55.7939 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:56.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:56.1760 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428147] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:56.3046 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:56.7630 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:56.7921 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:57.0769 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:57.1751 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:57.1761 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428148] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:57.6102 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:57.7115 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:57.8074 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:58.1766 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:15:58.1767 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428148] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:58.3173 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:58.5889 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): [ W 2019-10-07 09:15:58.5890 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428148] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:58.7762 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:58.8037 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:59.0903 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:59.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:15:59.1762 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428149] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:15:59.6231 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:59.7231 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:15:59.8198 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:00.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:00.1761 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428149] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:00.3310 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:00.7877 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:00.8138 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:01.1005 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:01.1759 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428150] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:01.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:01.6391 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:01.7298 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:01.8324 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:01.9892 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1blpt9ue933xp8kydkcplvy01p2meorof6l1atqhn9udsnaohe' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:01.9895 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428151] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:02.0058 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1blpt9ue933xp8kydkcplvy01p2meorof6l1atqhn9udsnaohe' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:02.0061 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428151] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:02.0929 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1ql2sp46vj8gf19mn84lodqsok6kuqto9jmhvkyr1ydqy38pd4' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:02.0932 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428151] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:02.0986 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1ql2sp46vj8gf19mn84lodqsok6kuqto9jmhvkyr1ydqy38pd4' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:02.0988 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428151] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:02.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:16:02.1762 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428151] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:02.3429 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:02.6812 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1twd101o0d4dzk9en4fsv5hi8q3g9zsb9lm4zy7e87a6kc4akf' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' [ W 2019-10-07 09:16:02.6813 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428151] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:02.7084 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428151] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:02.7082 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1twd101o0d4dzk9en4fsv5hi8q3g9zsb9lm4zy7e87a6kc4akf' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:02.7986 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:02.8246 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:03.1116 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:03.1753 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:16:03.1754 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428152] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:03.6529 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:03.7040 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' [ W 2019-10-07 09:16:03.7042 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428152] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:03.7420 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:03.8379 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:04.1759 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428153] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:04.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:04.3540 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:04.8102 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:04.8345 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:05.1237 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:05.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:16:05.1760 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428153] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:05.6698 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:05.7528 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:05.8482 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:06.1753 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:16:06.1754 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428154] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:06.3591 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:06.8235 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:06.8465 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:07.1353 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:07.1747 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:16:07.1748 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428154] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:07.6815 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:07.7627 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:07.8595 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:08.1766 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' [ W 2019-10-07 09:16:08.1768 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428155] Sending 502 response: application did not send a complete response App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:08.3763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:08.7588 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' [ W 2019-10-07 09:16:08.7590 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428155] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:08.8356 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:08.8510 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:09.1470 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:09.1748 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428156] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:09.1746 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:09.6901 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:09.7741 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:09.8712 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:10.1755 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428156] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:10.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:10.3875 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:10.8412 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:10.8606 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:11.1582 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:11.1747 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:11.1748 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428157] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:11.7031 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:11.7851 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:11.8821 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:12.1750 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:16:12.1752 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428157] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:12.3987 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:12.8532 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:12.8714 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:13.1692 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:13.1745 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:13.1747 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428158] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:13.7156 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:13.7970 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:13.8937 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:13.9324 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' [ W 2019-10-07 09:16:13.9326 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428158] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:14.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' [ W 2019-10-07 09:16:14.1757 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428158] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:14.4106 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:14.8647 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:14.8838 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:15.1776 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' [ W 2019-10-07 09:16:15.1777 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428159] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:15.1806 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:15.7322 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:15.8084 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:15.9041 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:16.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' [ W 2019-10-07 09:16:16.1759 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428160] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:16.4220 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:16.8774 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:16.8943 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:17.1760 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' [ W 2019-10-07 09:16:17.1762 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428160] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:17.1856 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:17.7464 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:17.8201 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:17.9218 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:18.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' [ W 2019-10-07 09:16:18.1765 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428161] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:18.4334 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:18.8886 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:18.9049 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:18.9923 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' [ W 2019-10-07 09:16:18.9925 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428161] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:19.1762 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:16:19.1764 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428161] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:19.1968 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:19.7579 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:19.8251 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:19.9340 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:20.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:16:20.1759 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428162] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:20.4454 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:20.8999 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:20.9206 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:21.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' [ W 2019-10-07 09:16:21.1761 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428162] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:21.2073 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:21.7717 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:21.8402 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:21.9403 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:22.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:16:22.1758 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428163] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:22.4564 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:22.9109 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:22.9321 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:23.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' [ W 2019-10-07 09:16:23.1758 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428164] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:23.2200 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:23.7858 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:23.8532 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:23.9530 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:24.1186 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): [ W 2019-10-07 09:16:24.1187 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428164] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:24.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' [ W 2019-10-07 09:16:24.1757 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428164] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:24.4618 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:24.9221 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:24.9493 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:25.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:16:25.1761 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428165] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:25.2318 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:25.7988 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:25.8645 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:25.9647 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:26.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:26.1759 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428165] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:26.4734 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:26.9342 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:26.9605 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:27.1764 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:16:27.1765 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428166] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:27.2427 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:27.8060 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:27.8763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:27.9761 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:28.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' [ W 2019-10-07 09:16:28.1756 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428167] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:28.4853 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:28.9400 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:28.9879 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:29.1762 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' [ W 2019-10-07 09:16:29.1764 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428167] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:29.2111 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' [ W 2019-10-07 09:16:29.2112 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428167] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:29.2549 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:29.8218 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:29.8877 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:29.9892 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:30.1759 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' [ W 2019-10-07 09:16:30.1760 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428168] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:30.4973 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:30.9519 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:31.0037 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:31.1758 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' [ W 2019-10-07 09:16:31.1760 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428168] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:31.2716 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:31.8405 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:31.9009 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:32.0018 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:32.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' [ W 2019-10-07 09:16:32.1758 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428169] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:32.5089 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:32.9639 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:33.0167 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:33.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' [ W 2019-10-07 09:16:33.1758 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428169] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:33.2850 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:33.8530 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:33.9133 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:34.0134 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:34.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' [ W 2019-10-07 09:16:34.1759 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428170] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:34.2556 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' [ W 2019-10-07 09:16:34.2557 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428170] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:34.5208 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:34.9754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:35.0276 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:35.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' [ W 2019-10-07 09:16:35.1757 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428171] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:35.2999 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:35.8674 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:35.9254 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:36.0261 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:36.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:36.1759 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428171] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:36.5323 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:36.9870 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:37.0404 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:37.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' [ W 2019-10-07 09:16:37.1757 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428172] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:37.3111 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:37.8817 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:37.9307 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:38.0380 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:38.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' [ W 2019-10-07 09:16:38.1757 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428172] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:38.5443 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:38.9994 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:39.0537 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:39.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' [ W 2019-10-07 09:16:39.1757 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428173] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:39.3246 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:39.3465 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:16:39.3467 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428173] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:39.8985 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:39.9453 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:40.0447 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:40.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:16:40.1756 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428174] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:40.5563 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:41.0105 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:41.0659 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:41.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:41.1758 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428174] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:41.3361 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:41.9103 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:41.9571 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:42.0570 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:42.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:42.1758 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428175] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:42.5621 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:43.0232 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:43.0785 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:43.1752 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:43.1754 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428175] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:43.3482 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:43.9247 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:43.9696 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:44.0723 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:44.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' [ W 2019-10-07 09:16:44.1756 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428176] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:44.4857 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' [ W 2019-10-07 09:16:44.4859 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428176] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:44.5732 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:45.0356 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:45.0840 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:45.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:45.1758 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428177] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:45.3653 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:45.9313 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:45.9811 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:46.0844 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:46.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:46.1758 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428177] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:46.5918 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:47.0416 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:47.0948 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:47.1751 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): [ W 2019-10-07 09:16:47.1752 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428178] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:47.3764 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:47.9467 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:47.9923 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:48.0960 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:48.1753 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:48.1756 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428178] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:48.6027 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:49.0533 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:49.1060 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:49.1755 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:16:49.1756 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428179] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:49.3876 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:49.4924 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' [ W 2019-10-07 09:16:49.4925 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428179] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:49.9611 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:50.0061 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:50.0532 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='5vqmz9mik2ou2k9objb8pnyce8t97p6vocyaouqo3qalvpmjs5' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' [ W 2019-10-07 09:16:50.0533 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428179] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:50.0549 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='5vqmz9mik2ou2k9objb8pnyce8t97p6vocyaouqo3qalvpmjs5' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): [ W 2019-10-07 09:16:50.0550 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428180] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:50.1086 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:50.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:50.1758 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428180] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:50.6130 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:51.0657 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:51.1163 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:51.1748 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:51.1750 125344/Tk age/Cor/Con/InternalUtils.cpp:96 ]: [Client 7-428180] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:51.4034 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:51.9729 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:52.0195 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:52.0660 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428181] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:52.0659 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='5vqmz9mik2ou2k9objb8pnyce8t97p6vocyaouqo3qalvpmjs5' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:52.1214 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:52.1757 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): [ W 2019-10-07 09:16:52.1758 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428181] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:52.6248 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:53.0827 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:53.1262 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:53.1756 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:53.1758 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428182] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:53.4090 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:53.9857 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:54.0308 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:54.0764 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='5vqmz9mik2ou2k9objb8pnyce8t97p6vocyaouqo3qalvpmjs5' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:54.0766 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428182] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:54.1341 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:54.1754 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:54.1757 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428182] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:54.6128 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:54.6132 125344/T8 age/Cor/Con/InternalUtils.cpp:96 ]: [Client 1-428183] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:54.6367 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:55.0943 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:55.1397 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:55.1751 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' [ W 2019-10-07 09:16:55.1753 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428183] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:55.4215 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:55.9983 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:56.0370 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:56.1465 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:56.1747 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:56.1750 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428184] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:56.6493 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:57.1050 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:57.1512 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:57.1746 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): [ W 2019-10-07 09:16:57.1747 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428184] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:57.4332 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:58.0125 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:58.0474 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:58.1534 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:58.1749 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:16:58.1750 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428185] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:58.6603 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:59.0810 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='5vqmz9mik2ou2k9objb8pnyce8t97p6vocyaouqo3qalvpmjs5' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' [ W 2019-10-07 09:16:59.0812 125344/Tg age/Cor/Con/InternalUtils.cpp:96 ]: [Client 5-428185] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:59.1162 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:59.1640 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:59.1746 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:16:59.1748 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428185] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:16:59.4451 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:16:59.6546 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:16:59.6547 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428186] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2bmbukj5jvkf4y7a5bj5rbvpkcuhm6smbpu3j2wtbzeahns3s3' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:00.0244 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:00.0631 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:00.1707 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:00.1750 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:17:00.1753 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428186] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:17:00.6657 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:01.0927 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='5vqmz9mik2ou2k9objb8pnyce8t97p6vocyaouqo3qalvpmjs5' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:17:01.0931 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428186] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:17:01.1270 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:01.1763 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:01.1988 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:17:01.1991 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428187] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:17:01.4557 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:02.0364 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:02.0734 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:02.1752 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' [ W 2019-10-07 09:17:02.1753 125344/Tm age/Cor/Con/InternalUtils.cpp:96 ]: [Client 8-428187] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:02.1829 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:02.6793 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:02.7832 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='1ql2sp46vj8gf19mn84lodqsok6kuqto9jmhvkyr1ydqy38pd4' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' [ W 2019-10-07 09:17:02.7834 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428188] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:03.1039 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='5vqmz9mik2ou2k9objb8pnyce8t97p6vocyaouqo3qalvpmjs5' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' [ W 2019-10-07 09:17:03.1041 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428188] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:03.1377 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:03.1795 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server [ W 2019-10-07 09:17:03.1796 125344/Ti age/Cor/Con/InternalUtils.cpp:96 ]: [Client 6-428188] Sending 502 response: application did not send a complete response App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:03.1880 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:03.4666 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:04.0439 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:04.0851 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:04.1753 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' [ W 2019-10-07 09:17:04.1755 125344/Tc age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-428189] Sending 502 response: application did not send a complete response App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125491 output: [ 2019-10-07 09:17:04.1952 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' App 125551 output: WARNING: terminating connection because of crash of another server process App 125551 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125551 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125571 output: WARNING: terminating connection because of crash of another server process App 125571 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125571 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125491 output: I, [2019-10-07T09:43:49.931737 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125511 output: I, [2019-10-07T09:49:09.705215 #125511] INFO -- omniauth: (joshid) Request phase initiated. App 125511 output: I, [2019-10-07T09:49:37.015999 #125511] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T09:53:52.325181 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125511 output: I, [2019-10-07T11:10:38.290267 #125511] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T11:11:18.019148 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T11:24:08.972104 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125591 output: WARNING: terminating connection because of crash of another server process App 125591 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125591 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125511 output: I, [2019-10-07T11:51:57.642161 #125511] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T12:11:03.511893 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T13:07:15.464780 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T13:31:58.024383 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125491 output: I, [2019-10-07T14:13:54.435587 #125491] INFO -- omniauth: (joshid) Request phase initiated. App 125611 output: WARNING: terminating connection because of crash of another server process App 125611 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125611 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125491 output: I, [2019-10-07T15:02:07.938763 #125491] INFO -- omniauth: (joshid) Request phase initiated.